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Summary REL2011-No.no 

The relationship between English 
proficiency and content knowledge 
for English language learner students 
in grades 10 and 11 in Utah 



This study examines the relationship 
between performance on Utah's Eng- 
lish proficiency assessment and English 
language arts and mathematics content 
assessments by English language learner 
students and compares the performance 
of English language learner and non- 
English language learner students on the 
content assessments. 

Utah has approximately 45,000 English lan- 
guage learner students (U.S. Department of 
Education 2009), or about 8 percent of its K-12 
student population. Staff in the Utah State 
Office of Education are interested in better 
understanding the academic performance of 
English language learner students in relation 
to their English proficiency and in compari- 
son with the academic performance of their 
non-English language learner peers. Thus, 
Utah State Office of Education staff asked 
Regional Educational Laboratory (REL) West 
to study the performance of English language 
learner students in grades 10 and 11 on the 
Utah Academic Language Proficiency Assess- 
ment (UALPA) and on the English language 
arts and mathematics content assessments 
administered under the Utah Performance As- 
sessment System for Students (U-PASS). They 
also asked that the study compare English 



language learner students’ and their non-Eng- 
lish language learner peers’ performance on 
the content assessments. 

Anecdotal reports of English language learner 
students who performed better on the content 
assessments than on the language proficiency 
assessment, a counterintuitive outcome, had 
raised questions about assessment valid- 
ity. Utah State Office of Education staff also 
wanted a comparison of English language 
learner students’ performance on the language 
proficiency assessment and their performance 
on the content assessments because Utah’s 
rules for redesignating English language 
learner students as fluent, which are based 
on performance on both assessments, had 
changed at the beginning of 2008/09, lowering 
the required U-PASS English language arts 
assessment score. And state staff had seen a 
performance gap between English language 
learner students’ and their non-English 
language learner peers and wanted to better 
understand its magnitude and any content- 
specific patterns. 

Two research questions guided this study: 

1. What is the distribution of performance 
of English language learner students in 



SUMMARY 



grades 10 and 11 on the Utah Academic 
Language Proficiency Assessment, com- 
pared with their performance on the 
English language arts and mathematics 
content assessments of the Utah Perfor- 
mance Assessment System for Students? 

2. How does the performance of English 
language learner students on the English 
language arts and mathematics content 
assessments of the Utah Performance 
Assessment System for Students compare 
with that of non-English language learner 
students, overall and by content standard? 

The Utah State Office of Education provided 
student-level datasets of all grade 10 and 11 
students who took the language proficiency 
and content assessments in 2008/09. The 
language proficiency datasets included overall 
scaled scores and score categories (pre-emer- 
gent, emergent, intermediate, and advanced) 
and the content assessment datasets included 
the number and percent of correct responses 
by content standard for each student. Student 
records were linked across assessments by an 
anonymous numerical identifier. 

Findings include: 

• Of grade 10 students who scored below 

intermediate on the language proficiency 
assessment, 6 percent scored 3 or 4 (the 
highest two scores) on the English lan- 
guage arts content assessment; of grade 10 
students who scored intermediate on the 
language proficiency assessment, 14 per- 
cent scored 3 or 4 on the English language 
arts content assessment; and of grade 10 
students who scored advanced on the lan- 
guage proficiency assessment, 59 percent 



scored 3 or 4 on the English language arts 
content assessment. 

• Of grade 11 students who scored below 
intermediate on the language proficiency 
assessment, 1 percent scored 3 or 4 on 
the English language arts content assess- 
ment; of grade 11 students who scored 
intermediate on the language proficiency 
assessment, 9 percent scored 3 or 4 on 
the English language arts content as- 
sessment; and of grade 11 students who 
scored advanced on the language profi- 
ciency assessment, 33 percent scored 3 
or 4 on the English language arts content 
assessment. 

• The pre-2008/09 criteria for redesignating 
students as fluent in English were met by 
19.7 percent of English language learner 
students in grade 10 and 12.9 percent 

in grade 11. The criteria introduced in 
2008/09 were met by 25.4 percent of Eng- 
lish language learner students in grade 10 
and 19.5 percent in grade 11. 

• About 9 percent of English language 
learner students in grade 10 and 5 percent 
in grade 11 scored 3 or 4 on the English 
language arts content assessment and 
below advanced on the language profi- 
ciency assessment. 

• English language learner students were 
outscored by non-English language 
learner students by an average of 1.3-1. 6 
standard deviations in both grades on the 
English language arts content assessment 
and by an average of 0.4-0.7 standard 
deviation on the mathematics content 
assessments. 



SUMMARY 



With the continuing growth over the past 
decade in the number of English language 
learner students in the West Region, help- 
ing states better serve this subgroup is a 
priority for REL West. The information from 
this study is expected to aid the Utah State 
Office of Education in generating descrip- 
tive information about English language 
learner students’ performance on the lan- 
guage proficiency and content assessments, 



in informing ongoing discussions of rules 
related to when students should be moved 
out of English language learner status, and 
in helping the state agency’s curriculum 
and instruction and assessment programs 
develop a richer understanding of English 
language learner students’ performance on 
specific content. 

March 2011 
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This study 
examines the 
relationship 
between 
performance on 
Utah's English 
proficiency 
assessment 
and English 
language arts 
and mathematics 
content 

assessments by 
English language 
learner students 
and compares the 
performance of 
English language 
learner and non- 
English language 
learner students 
on the content 
assessments. 



WHY THIS STUDY? 

With the continued growth over the past decade in 
the number of English language learner students 
in the West Region, helping states better serve this 
subgroup is a priority for Regional Educational 
Laboratory (REL) West. Utah has approximately 
45,000 English language learner students (U.S. 
Department of Education 2009), or about 8 percent 
of its K-12 student population (see box 1 for more 
on English language learner students in Utah). 
Staff from the Utah State Office of Education are 
interested in better understanding the academic 
performance of students in this subgroup in rela- 
tion to their English proficiency and in compari- 
son with the academic performance of their non- 
English language learner peers. Agency staff asked 
REL West to explore the relationship between the 
performance of Utah’s English language learner 
students on the state’s language proficiency assess- 
ment, the Utah Academic Language Proficiency 
Assessment (UALPA), and their performance on 
the state’s English language arts and mathemat- 
ics content assessments, which are administered 
under the Utah Performance Assessment System 
for Students (U-PASS; see box 2 for more informa- 
tion on these assessments). 

Utah’s request was based in part on anecdotal 
reports of high school English language learner 
students who scored below advanced on the state’s 
language proficiency assessment (the highest score 
and one of the requirements for redesignating 
English language learner students as fluent) but 
scored 3 or 4 (the highest two scores and the other 
requirement for redesignating English language 
learner students as fluent prior to 2008/09; see 
box 1) on the state content assessment for English 
language arts. This outcome runs counter to the 
assumption that English language proficiency as 
measured by the UALPA was an accurate indicator 
of the ability to understand content and thus to 
perform well on content assessments, particularly 
the English language arts assessment (D. Smith, 
former Utah Assessment Development Coordi- 
nator, personal communication, December 16, 
2008). This seemingly anomalous score pairing 
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BOX 1 

Determining English language 
learner status in Utah 

In Utah, all newly enrolled students 
in a district are given a home- 
language survey to determine 
whether they should be considered 
for English language learner identi- 
fication. Those who have a primary 
home language other than English 
take the Utah Academic Language 
Proficiency Assessment (UALPA) to 
assess their English proficiency (pre- 
emergent, emergent, intermediate, 
or advanced; see box 2) and deter- 
mine whether to classify them as an 
English language learner student. If a 
student is classified as an English lan- 
guage learner, the school’s Alterna- 
tive Language Services team meets to 
decide how to best assist the student 
in acquiring English proficiency. 
Parents are notified, and unless they 
opt otherwise, the student begins to 
receive language support services as 
soon as possible. 

BOX 2 

About the Utah Academic 
Language Proficiency Assessment 
and the Utah Performance 
Assessment System for Students 

Two assessments are used in this 
study: the Utah Academic Language 
Proficiency Assessment (UALPA) and 
the Utah Performance Assessment 
System for Students (U-PASS). 

Utah Academic Language Proficiency 
Assessment. The UALPA is admin- 
istered annually to every English 
language learner student enrolled in 



The official definition of an English 
language learner student, accord- 
ing to Rule R277-716, Alternative 
Language Services for Utah Students 
(Utah Department of Administrative 
Services n.d.), is an individual: 

“who has sufficient difficulty speak- 
ing, reading, and writing or under- 
standing the English language and 
whose difficulties may deny such 
individual the opportunity to learn 
successfully in classrooms where the 
language of instruction is English or 
which may deny the individual the 
opportunity to participate fully in 
society; or 

“who was not born in the United 
States or whose native language is a 
language other than English and who 
comes from an environment where 
a language other than English is 
dominant; or 

“who is an American Indian or Alas- 
kan native or who is a native resident 



a Utah district prior to April 15 of the 
school year. It measures English lan- 
guage proficiency in the four language 
acquisition modalities— listening, 
speaking, reading, and writing — and 
yields a comprehension score calcu- 
lated from selected parts of the read- 
ing and listening scores. The UALPA 
identifies students as performing at 
one of four English proficiency levels: 
pre-emergent, emergent, intermedi- 
ate, and advanced (Utah State Office 
of Education 2007): 

“Pre-emergent (P): The student has 
limited or no understanding of oral 



of the outlying areas and comes from 
an environment where a language 
other than English has had a signifi- 
cant impact on such individual’s level 
of English language proficiency.” 

Scores from the UALPA and the 
Utah Performance Assessment 
System for Students (U-PASS) are 
used in determining when to reclas- 
sify a student in English language 
learner status to fluent status. In 
2006/07 and 2007/08, the first two 
years the UALPA was administered, 
a score of advanced on the UALPA, 
together with a score of 3 or 4 (the 
highest two scores) on the English 
language arts assessment of the 
U-PASS (see box 2), was required for 
an English language learner student 
to be redesignated as fluent. Begin- 
ning in 2008/09, a score of advanced 
on the UALPA together with a 
score of 2b (the third highest score) 
on the U-PASS English language 
arts assessment was sufficient for 
redesignation. 

or written English, so he/she partici- 
pates by listening. The student may 
demonstrate comprehension by using 
a few isolated words or expressions of 
speech. The student typically draws, 
copies, or responds verbally in his/ 
her native language to simple com- 
mands, statements and questions. 

The student may understand the 
relationship between oral and written 
languages. 

“Emergent (E): The student has begun 
to understand that written language 
represents oral language. The student 
understands and responds to basic 
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BOX 2 (CONTINUED) 

About the Utah Academic Language Proficiency Assessment and the Utah Performance Assessment 
System for Students 



social conventions, simple questions, 
simple directions, and appropri- 
ate level text. He/she participates in 
classroom routines. In general, the 
student speaks, reads, and writes 
using single words, short phrases, or 
simple sentences with support. Over- 
all, the student has minimal expres- 
sive vocabulary. 

“Intermediate (I): The student un- 
derstands and uses more abstract, 
academic, and formal language and 
literacy skills. He/she participates 
actively in most social and class- 
room tasks using idioms and more 
content-specific language in speech 
and writing. The student reads and 
writes independently for personal 
and academic purposes, with some 
persistent errors. With some support, 
the student reads and writes about 
various topics using different genres 
for a variety of audiences. 

“Advanced (A): The student has devel- 
oped proficiency in the English lan- 
guage and literacy skills. The student 
may need continued support when 
engaged in complex academic tasks 
that require increasingly academic 



language. Students at this level attend 
mainstream or sheltered instruction 
classes.” 

All newly enrolled English language 
learner students and all English 
language learner students who score 
below advanced on the UALPA are re- 
quired to take the assessment during 
its next administration (Utah State 
Office of Education 2009; see box 1). 

Utah Performance Assessment System 
for Students. U-PASS, Utah’s as- 
sessment and school accountability 
system, tests students using a crite- 
rion-referenced content assessment — 
annually in grades 2-11 for English 
language arts and annually in grades 
2-7, with assessments by course 
thereafter, for mathematics. 

The U-PASS content standards 
examined for grade 10 and 11 
English language arts are standard 
I (comprehend informational and 
literary text), standard II (write 
informational and literary text), and 
standard III (seek and give informa- 
tion using the process of inquiry). 

The content standards examined 



for algebra I are standard I (expand 
number sense to understand, per- 
form operations, and solve problems 
with real numbers), standard II 
(extend concepts of proportion to 
represent and analyze linear rela- 
tions), standard III (develop fluency 
with the language and operations 
of algebra to analyze and represent 
relationships), and standard IV 
(understand concepts from statistics 
and apply statistical methods to solve 
problems). The content standards 
examined for geometry are standard 
I (use algebraic, spatial, and logical 
reasoning to solve geometry prob- 
lems), standard II (use the language 
and operations of algebra to explore 
geometric relationships with coordi- 
nate geometry), standard III (extend 
concepts of proportion and simi- 
larity to trigonometric ratios), and 
standard IV (use algebraic, spatial, 
and logical reasoning to solve mea- 
surement problems). 

U-PASS performance is split into six 
score categories, la, lb, 2a, 2b, 3, and 
4 in order from lowest to highest per- 
formance. (The state does not apply 
text labels to these categories.) 



raised questions about the validity of the language 
proficiency assessment, and Utah State Office of 
Education staff wanted to know the prevalence of 
this unexpected combination. 

The purposes of the assessments are different: the 
language proficiency assessment tests proficiency 
in listening, speaking, reading, and writing the 
English language. The content assessment in Eng- 
lish language arts measures mastery of academic 



content and achievement standards in English 
language arts. Furthermore, neither assessment 
was developed taking into account the properties 
of the other, so the meaning of a student’s score 
on one assessment is unclear in the context of the 
student’s score on the other. 

Better understanding connections between 
language proficiency and academic proficiency is 
also important in Utah’s continuing effort to set 
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Better understanding 
connections between 
language proficiency and 
academic proficiency 
is important in Utah's 
continuing effort to set 
the bar appropriately 
for redesignating 
English language learner 
students as fluent and 
removing them from 
programs and services 
supporting students 
who are not yet fluent 



the bar appropriately for redesig- 
nating English language learner 
students as fluent and removing 
them from programs and ser- 
vices supporting students who 
are not yet fluent. Scores from 
both assessments have provided 
evidence for determining whether 
to reclassify individual English 
language learner students. At the 
beginning of the 2008/09 school 
year, the U-PASS performance 
requirement was changed so 
that a lower score on the English 
language arts assessment, along 
with an advanced score on the 
UALPA, allowed a student to be redesignated (see 
box 1). State staff wanted to know how this change 
affected the number and percentage of English 
language learner students who would be redes- 
ignated. Because exit rules for English language 
learner status are likely to continue to include a 
baseline achievement level on both the language 
proficiency assessment and the English language 
arts content assessment, understanding the 
relationship between performance on the UALPA 
and the U-PASS content assessments is helpful 
to developing and maintaining sound English lan- 
guage learner student policy. 



There are three reasons behind the study’s focus 
on the assessment performance of students in 
grades 10 and 11 and not other grades. First, the 
U-PASS English language arts assessment shows 
a large performance gap between English lan- 
guage learner students and non-English language 
learner students at the secondary level. In par- 
ticular, whereas at the elementary level, English 
language learner students who score advanced 
on the UALPA score 3 or 4 on U-PASS at rates 
comparable to those of non-English language 
learner students, at the secondary level, English 
language learner students who score advanced on 
the UALPA trail other students in scoring 3 or 4 
on U-PASS by 12 percentage points or more (R. 
Honaker, former Utah Assessment Results Coor- 
dinator, and D. Smith, personal communication, 



December 16, 2008). Second, Utah State Office of 
Education staff have identified the gap between 
these two groups of students at the secondary level 
as a topic of particular concern (R. Honaker and 
D. Smith, personal communication, December 16, 
2008). Third, certain data do not exist for students 
in grade 12 (for example, students do not take the 
U-PASS content assessment in English language 
arts in grade 12), but enough students take the 
UALPA and the U-PASS in grades 10 and 11. 

State Office of Education staff also sought to better 
understand the achievement gap between English 
language learner students and their non-English 
language learner peers. Although the Utah State 
Office of Education reports the percentage of 
students, both overall and by English language 
learner status, who score 3 or 4 (the highest two 
scores) on the U-PASS, its staff have neither 
analyzed the differences in terms of scaled score 
points nor examined group performance differ- 
ences by content standard. Such analysis could 
improve the agency’s curriculum and instruction 
experts’ understanding of relationships between 
language proficiency status and specific content. 
English language arts content standards are the 
same for grades 10 and 11: standard I addresses 
comprehension, standard II addresses writing, 
and standard III addresses inquiry. Mathematics 
standards are unique to each subject, so there is 
no crossover between standard I for algebra I and 
standard I for geometry, for example (see box 2). 
Box 3 provides more information on the data and 
methodology behind this study. 

The findings of this study are expected to aid the 
Utah State Office of Education in generating basic 
descriptive information about English language 
learner students’ performance on the language 
proficiency and content assessments, in inform- 
ing ongoing discussions of rules related to when 
students should be moved out of official Eng- 
lish language learner status, and in helping the 
agency’s curriculum and instruction and assess- 
ment programs develop a richer understanding of 
English language learner students’ performance 
on specific content. 
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BOX 3 

Data and methodology 

Data. Regional Educational Labora- 
tory (REL) West acquired assessment 
and demographic data files from the 
Utah State Office of Education. Each 
file was stripped of information that 
could identify individual students. 
Numerical student identifiers allowed 
the research team to link student 
records across assessment and demo- 
graphic data files. 

The dataset included 36,326 students 
in grade 10 (99.3 percent of the state’s 
36,584 grade 10 students) and 34,837 
students in grade 11 (99.1 percent of 
the state’s 35,146 grade 11 students) 
who took the Utah Performance 
Assessment System for Students (U- 
PASS) English language arts assess- 
ment, 9,590 students in grades 10 and 
11 who took the algebra I assessment, 
and 17,112 students in grades 10 and 
11 who took the geometry assessment 
in 2008/09. For comparisons of sub- 
group performance on the U-PASS 
content assessments by content stan- 
dard, student-level data that included 
the raw score on each standard was 
provided. 

In 2008/09, 3,089 (4.3 percent) of the 
71,730 students in grades 10 and 11 
who are included in the analysis were 
classified as English language learner 
students; 76 percent were Elispanic, 

7 percent were Asian, 5 percent 
were Pacific Islander, 4 percent were 
American Indian, 4 percent were 
White, and 3 percent were Black (less 
than 1 percent did not indicate an 
ethnicity). In addition, 17 percent 
were students with disabilities, and 



73 percent received free or reduced- 
price meals. And 2,629 (85.1 percent) 
of the English language learners 
in this study had a Utah Academic 
Language Proficiency Assessment 
(UALPA) score for 2008/09; 2 percent 
of them were categorized as pre- 
emergent, 8 percent as emergent, 

55 percent as intermediate, and 35 
percent as advanced. 

Non-English language learner 
students accounted for 95.7 percent 
of Utah students in grades 10 and 11. 
Of the non-English language learner 
students in the study, 84 percent were 
White, 10 percent were Hispanic, 2 
percent were Asian, 1 percent were 
Pacific Islander, 1 percent were 
American Indian, and 1 percent were 
Black (less than 1 percent did not 
indicate an ethnicity). In addition, 8 
percent were students with disabili- 
ties, and 24 percent received free or 
reduced-price meals. 

Methodology. The joint distribution of 
UALPA and U-PASS performance is 
summarized in grade-specific cross- 
tabulations. A 3x6 cross-tabulation 
for each grade and subject summa- 
rizes joint performance in terms of 
performance categories. Although the 
UALPA has four score categories, the 
pre-emergent and emergent catego- 
ries are sparsely populated and were 
combined for this study. The cross- 
tabulations also inform the question 
of rules for exiting English language 
learner status. The percentage of 
students meeting various conjunc- 
tive rules — those involving perfor- 
mance on both assessments — can 
be read from cells in the appropriate 
table, and rules based on a single 



assessment can be read from row or 
column totals. To address the rela- 
tionship between UALPA and U-PASS 
scores, a 2x2 contingency table for 
each subject and grade is presented. 

The distribution of scaled scores on 
the U-PASS is shown in box-and- 
whisker plots for English language 
learner and non-English language 
learner students. To report group 
performance differences by content 
standard, both raw score differences 
and normalized differences are pre- 
sented. Normalized differences are 
raw score differences divided by the 
pooled (English language learner and 
non-English language learner stu- 
dents combined) standard deviation 
of scores on each content standard. 
The pooled variance is 

Spooled = [C^ELL ~ -O^ELlT + C^NON ~ 1) 

®NON ] I C^ELL + ^NON _ 2) 

where ELL denotes the English lan- 
guage learner group and NON denotes 
the non-English language learner 
group. Comparing raw scores alone 
lacks a common metric and may be 
misleading. Because content-level 
scores are based on varying numbers 
of items, differences between the 
scores of English language learner 
students and those of non-English 
language learner students can be 
better understood by reporting both 
normalized and raw differences. 

In statewide assessments in other 
states, content standard scores 
for individuals can be based on 
seven or fewer items; in such cases, 
reliability is a concern (Skorupski 
2008). However, Utah’s grade 10 
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BOX 3 (CONTINUED) 

Data and methodology 

and 11 content assessments include 
at least 10 items for each standard. 
Moreover, the current study’s 
comparisons of performance at the 
content standard level are aver- 
ages over hundreds of individuals 
in the case of English language 



learner students and thousands 
of individuals in the case of non- 
English language learner students, 
making questions of the reliability 
of an individual’s content stan- 
dard score irrelevant. However, to 
provide more information about 



the differences between the perfor- 
mance of English language learner 
students and non-English language 
learner counterparts, both absolute 
and normalized differences are 
reported. 



Two research questions guide this study: 

1. What is the distribution of performance of 
English language learner students in grades 
10 and 11 on the Utah Academic Language 
Proficiency Assessment, compared with their 
performance on the English language arts 
and mathematics content assessments of the 
Utah Performance Assessment System for 
Students? 

• What is the joint distribution of English 
language learner students across the 
performance categories of the UALPA and 
the U-PASS? 

• What was the effect of changing the 
redesignation rules related to scores 
on the U-PASS English language 
arts assessment and the UALPA on 
the percentage of English language 
learner students meeting redesignation 
requirements? 

• How do advanced scores on the UALPA 
relate to scores of 3 or 4 (the highest two 
scores) on the U-PASS English language 
arts and mathematics assessments for 
English language learner students? 

2. How does the performance of English 
language learner students on the English lan- 
guage arts and mathematics content assess- 
ments of the Utah Performance Assessment 



System for Students compare with that of 
non-English language learner students, over- 
all and by content standard? 

• What is the overall distribution of 
U-PASS performance by English language 
learner status? 

• How does the distribution of English 
language learner students across perfor- 
mance categories on the U-PASS English 
language arts and mathematics assess- 
ments compare with that of non-English 
language learner students? 

• What are the differences in performance 
between English language learner stu- 
dents and non-English language learner 
students on the U-PASS English language 
arts and mathematics assessments by 
content standard? 

• What are the findings when English 
language learner students who scored ad- 
vanced on the UALPA are compared with 
non-English language learner students? 



This section discusses the findings related to the 
study’s two research questions on grade 10 and 
11 English language learner and non-English 
language learner students. 
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Comparing English language learner students' 
performance on the Utah Academic Language 
Proficiency Assessment and English language 
arts and mathematics content assessments 

Three subquestions were used to address the first 
research question. 

What is the joint distribution of English language 
learner students across the performance categories 
of the UALPA and the U-PASS? There is a positive 
association between English language learner 
students’ performance on the UALPA and their 
performance on the U-PASS English language arts 
assessment. Of the 9.6 percent of grade 10 English 



language learner students who scored pre-emer- 
gent or emergent on the UALPA, 6 percent scored 
3 or 4 (the highest two scores) on the U-PASS Eng- 
lish language arts assessment (table 1, figure 1). 

Of the 57.0 percent of grade 10 English language 
learner students who scored intermediate on the 
UALPA, 14 percent scored 3 or 4 on the U-PASS 
English language arts assessment. Of the 33.4 per- 
cent of grade 10 English language learner students 
who scored advanced on the UALPA, 59 percent 
scored 3 or 4 on the U-PASS English language arts 
assessment. 

The percentages of grade 11 English language 
learner students scoring in each UALPA category 



TABLE 1 

Joint distribution of English language learner students' scores on the Utah Academic Language Proficiency 
Assessment and the Utah Performance Assessment System for Students English language arts assessment, 
by grade level, 2008/09 (percent) 









UALPA category 




Grade and U-PASS 
English language arts score 3 


Pre-emergent 
or emergent 11 


Intermediate 


Advanced 


Total 


Grade 10 


la 


6.2 


12.7 


2.1 


21.0 


(N = 1,460) 


1b 


2.3 


16.2 


2.3 


20.9 




2a 


0.3 


12.3 


3.6 


16.2 




2b 


0.1 


7.5 


5.7 


13.3 




3 


0.5 


8.0 


16.3 


24.9 




4 


0.1 


0.1 


3.4 


3.7 




Total 


9.6 


57.0 


33.4 


100.0 




Percent of UALPA 
category scoring 3 or 4 


6 


14 


59 


29 


Grade 11 


la 


5.2 


12.7 


5.4 


23.4 


(A/ = 1,068) 


1b 


2.1 


20.2 


7.0 


29.3 




2a 


0.7 


9.9 


6.9 


17.5 




2b 


0.6 


4.9 


6.6 


12.0 




3 


0.1 


4.4 


11.0 


15.5 




4 


0.0 


0.4 


1.9 


2.2 




Total 


8.6 


52.5 


38.9 


100.0 




Percent of UALPA 
category scoring 3 or 4 


1 


9 


33 


18 



a. Higher numbers indicate better performance. 

b. The pre-emergent and emergent categories were combined because of the small number of students in each grade who were assessed at each category. 
Note.' Values may not sum to totals because of rounding. 

Source: Authors' analysis of Utah Academic Language Proficiency Assessment and Utah Performance Assessment System for Students data from the Utah 
State Office of Education. 




GRADE 11 I GRADE 10 
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FIGURE 1 

Distribution of English language learner students' 
scores on the Utah Performance Assessment System 
for Students English language arts assessment, 
by grade level and Utah Academic Language 
Proficiency Assessment score, 2008/09 (percent) 



FIGURE 2 

Distribution of English language learner students' 
scores on the Utah Performance Assessment 
System for Students mathematics assessments, by 
mathematics course and Utah Academic Language 
Proficiency Assessment score, 2008/09 (percent) 



U-PASS score® 

■ la ■ 1b 2a 2b 3 ■ 4 



Pre-emergent or 
emergent 11 ( N = 140) 

Intermediate (A/ = 832) 

Advanced (A/ = 488) 

Pre-emergent or 
emergent 11 (N = 92) 

Intermediate ( N = 561 ) 
Advanced (A/ = 41 5) 




0 25 50 75 100 



U-PASS score® 

■ la ■ 1b 2a 2b 3 ■ 4 




0 25 50 75 100 



Percent 



Percent 



a. Higher numbers indicate better performance. 

b. The pre-emergent and emergent categories were combined because 
of the small number of students in each grade who were assessed at 
each category. 

Note: Values may not sum to totals because of rounding. 

Source: Authors' analysis of Utah Academic Language Proficiency As- 
sessment and Utah Performance Assessment System for Students data 
from the Utah State Office of Education. 



a. Higher numbers indicate better performance. 

b. The pre-emergent and emergent categories were combined because 
of the small number of students in each grade who were assessed at 
each category. 

Note: Values may not sum to totals because of rounding. 

Source: Authors' analysis of Utah Academic Language Proficiency As- 
sessment and Utah Performance Assessment System for Students data 
from the Utah State Office of Education. 



are similar to the percentages of grade 10 students: 
8.6 percent scored pre-emergent or emergent, 

52.5 percent scored intermediate, and 38.9 per- 
cent scored advanced (see table 1). But a smaller 
percentage of grade 11 English language learner 
students in all UALPA categories scored 3 or 4 
(the highest two scores) on the U-PASS English 
language arts assessment. Of grade 11 English lan- 
guage learner students who scored pre-emergent 
or emergent on the UALPA, 1 percent scored 3 or 4 
on the U-PASS English language arts assessment. 
Of grade 11 English language learner students 
who scored intermediate on the UALPA, 9 percent 
scored 3 or 4 on the U-PASS English language arts 
assessment. And of grade 11 English language 
learner students who scored advanced on the 
UALPA, 33 percent scored 3 or 4 on the U-PASS 
English language arts assessment. 



There was also a positive association between 
English language learner students’ performance 
on the UALPA and their performance on the 
U-PASS mathematics assessments. Of the 28.7 
percent of English language learner students who 
took the U-PASS algebra I assessment and scored 
advanced on the UALPA, about 13 percent scored 
3 or 4 (the highest two scores) on the algebra I 
assessment (figure 2, table 2). And of the 45.5 
percent of English language learner students who 
took the U-PASS geometry assessment and scored 
advanced on the UALPA, 23 percent scored 3 or 4 
on the geometry assessment. 

What effect did changing the required scores on 
the U-PASS English language arts assessment have 
on the percentage of English language learner stu- 
dents meeting redesignation requirements? As of 
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TABLE 2 

Joint distribution of English language learner students' scores on the Utah Academic Language Proficiency 
Assessment and the Utah Performance Assessment System for Students mathematics assessments, by 
mathematics course, 2008/09 (percent) 



U-PASS English language arts score 3 




UALPA category 




Pre-emergent 
or emergent 13 


Intermediate 


Advanced 


Total 


Algebra 1 


la 


9.8 


34.5 


12.2 


56.5 


(N = 1,045) 


lb 


2.0 


9.7 


5.0 


16.7 




2a 


1.5 


7.6 


5.6 


14.6 




2b 


0.5 


3.0 


2.4 


5.8 




3 


0.5 


1.9 


3.3 


5.6 




4 


0.0 


0.4 


0.4 


0.8 




Total 


14.3 


57.0 


28.7 


100.0 




Percent of UALPA 












category scoring 3 or 4 


3 


4 


13 


6 


Geometry 


la 


3.3 


28.9 


19.9 


52.1 


(N = 792) 


1b 


1.0 


10.0 


7.6 


18.6 




2a 


0.0 


2.7 


3.3 


5.9 




2b 


0.0 


3.4 


4.3 


7.7 




3 


0.6 


3.9 


7.8 


12.4 




4 


0.0 


0.8 


2.5 


3.3 




Total 


4.9 


49.6 


45.5 


100.0 




Percent of UALPA 












category scoring 3 or 4 


13 


10 


23 


16 



a. Higher numbers indicate better performance. 

b. The pre-emergent and emergent categories were combined because of the small number of students in each grade who were assessed at each category. 
Note: Values may not sum to totals because of rounding. 

Source: Authors' analysis of Utah Academic Language Proficiency Assessment and Utah Performance Assessment System for Students data from the Utah 
State Office of Education. 



2008/09, English language learner students have 
to score advanced on the UALPA and score 2b, 3, 
or 4 on the U-PASS English language arts assess- 
ment in order to exit English language learner sta- 
tus; prior to 2008/09, a score of 2b on the U-PASS 
English language arts assessment was not suf- 
ficient to exit English language learner status (see 
box 1). Among English language learner students 
in grade 10, 19.7 percent had the U-PASS and 
UALPA scores necessary for redesignation under 
the old rule, and 25.4 percent did under the new 
rule, a gain of 5.7 percentage points or 29 percent 
(see table 1). Among English language learner 
students in grade 11, 12.9 percent had the U-PASS 
and UALPA scores necessary for redesignation 



under the old rule, and 19.5 percent did under 
the new rule, a gain of 6.6 percentage points or 51 
percent. 

How do advanced scores on the UALPA relate 
to scores of 3 or 4 (the highest two scores) on the 
U-PASS English language arts and mathematics 
assessments for English language learner students? 

While the joint distribution reported in the first 
subquestion provides information about the 
complete set of pairs of performance scores on the 
UALPA and the U-PASS, some combinations of 
scores are of more interest to policymakers. Specif- 
ically, accountability policy in Utah distinguishes 
between scoring advanced or not advanced 
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(pre-emergent, emergent, or intermediate) on 
UALPA and proficient (3 or 4) or not proficient (la, 
lb, 2a, or 2b) on the U-PASS. This section reports 
how an advanced score on the UALPA relates to a 
proficient score on the U-PASS. 

Some 57.7 percent of English language learner 
students in grade 10 and 56.3 percent in grade 
11 did not score advanced on the UALPA and 
did not score proficient on the U-PASS English 
language arts assessment (table 3). At the other 
end of the performance spectrum, 19.7 percent 
of English language learner students in grade 10 
and 12.9 percent in grade 11 scored advanced on 
the UALPA and proficient on the U-PASS English 
language arts assessment. Some 13.7 of Eng- 
lish language learner students in grade 10 and 
25.9 percent in grade 11 scored advanced on the 
UALPA but did not score proficient on the U-PASS 
assessment. And 8.8 percent of English language 
learner students in grade 10 and 4.9 percent in 
grade 11 did not score advanced on the UALPA but 



scored proficient on the U-PASS English language 
arts assessment. 

On the mathematics assessments, 68.5 percent of 
English language learner students who took the 
algebra I assessment and 49.2 percent who took the 
geometry assessment did not score advanced on the 
UALPA and did not score proficient on the U-PASS 
assessment (see table 3). By contrast, 3.6 percent 
of English language learner students who took the 
algebra I assessment and 10.4 percent who took 
the geometry assessment scored advanced on the 
UALPA and proficient on the U-PASS assessment. 
Some 25.1 percent of the English language learner 
students who took the algebra I assessment and 35.1 
who took the geometry assessment scored advanced 
on the UALPA but did not score proficient on the 
U-PASS assessment. And 2.8 percent of English 
language learner students who took the algebra I 
assessment and 5.3 percent who took the geometry 
assessment did not score advanced on the UALPA 
but scored proficient on the U-PASS assessment. 



TABLE 3 

Distribution of English language learner students, by Utah Academic Language Proficiency Assessment and 
Utah Performance Assessment System for Students scores and grade level, 2008/09 (percent) 









UALPA 




U-PASS content 


Grade 10 (N = 


1,460) 


Grade 11 (N = 


1,068) 


area and score 


Not advanced 3 


Advanced 


Not advanced 3 


Advanced 


English language arts 


Not proficient 13 


57.7 


13.7 


56.3 


25.9 


Proficient 13 


8.8 


19.7 


4.9 


12.9 


Total 


66.6 


33.4 


61.1 


38.9 




Algebra 1 (A/ = 


1,045) 


Geometry ( N 


= 792) 




Not advanced 3 


Advanced 


Not advanced 3 


Advanced 


Mathematics 


Not proficient 13 


68.5 


25.1 


49.2 


35.1 


Proficient 13 


2.8 


3.6 


5.3 


10.4 


Total 


71.3 


28.7 


54.5 


45.5 



a. Score of pre-emergent, emergent, or intermediate. 

b. Score of la, 1b, 2a, or 2b. 

c. Score of 3 or 4. 

Note: Values may not sum to totals because of rounding. 

Source: Authors' analysis of Utah Academic Language Proficiency Assessment and Utah Performance Assessment System for Students data from the Utah 
State Office of Education. 
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Comparing the performance of English language 
learner and non-English language learner students 
on English language arts and mathematics content 
assessments, overall and by content standard 

Four subquestions were used to address the second 
research question. 

What is the overall distribution ofU-PASS per- 
formance by English language learner status? The 

middle 50 percent of the distribution of scaled 
scores on the U-PASS English language arts 
assessment in both grades 10 and 11 was lower 
for English language learner students than for 
non-English language learner students, and the 
median score was 15-16 scaled score points lower 
(figure 3). In addition, more than 75 percent of the 
English language learner students scored below 
the 25th percentile of the non-English language 
learner students. 



The distribution of scaled scores on the U-PASS 
mathematics assessments for grades 10 and 11 
shows more overlap between English language 
learner students and non-English language 
learner students (figure 4). For both algebra I and 
geometry, the median score of English language 
learner students was 9 scaled score points lower 
than that of English language learner students, 
and more than 75 percent of English language 
learner students scored below the 50th percentile 
of non-English language learner students. 

How does the distribution of English language 
learner students across performance categories on 
the U-PASS English language arts and mathemat- 
ics assessments compare with that of non-English 
language learner students? The percentages of 
English language learner students who scored 3 or 
4 (the highest two scores) on the U-PASS English 
language arts assessment were lower than the 



FIGURE 3 

Distribution of scaled scores on the Utah 
Performance Assessment System for Students 
English language arts assessment, by grade level 
and English language learner status, 2008/09 



FIGURE 4 

Distribution of scaled scores on the Utah 
Performance Assessment System for Students 
mathematics assessments, by mathematics course 
and English language learner status, 2008/09 



English language arts scaled scores 



200 

190 

180 

170 

160 

150 

140 

130 




observation 



English language Non-English English language Non-English 
learner language learner learner language learner 
students students students students 

(A/ =1,691) {N =34,635) (N =1,284) (A/ =33,553) 



GRADE 10 



GRADE 11 



Note.- The scale score ranges from 130 to 199. 

Source: Authors' analysis of student demographic data and Utah Perfor- 
mance Assessment System for Students data from the Utah State Office 
of Education. 



Math scaled scores 



200 
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170 

160 
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140 

130 




observation 



English language Non-English English language Non-English 
learner language learner learner language learner 
students students students students 

(A/ =1,183) (A/ =8,407) [N =898) [N =16,214) 



ALGEBRA I 



GEOMETRY 



Note: Counts are greater for geometry because most grade 10 and 
11 non-English language learner students taking mathematics have 
already taken algebra I. The scale score ranges from 130 to 199. 

Source: Authors' analysis of student demographic data and Utah Perfor- 
mance Assessment System for Students data from the Utah State Office 
of Education. 
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percentages of non-English language learner stu- 
dents who did (figure 5). Among English language 
learner students, 28 percent in grade 10 and 20 
percent in grade 11 scored 3 or 4, while among 
non-English language learner students, 86 percent 
in grade 10 and 83 percent in grade 11 did. 

The percentages of English language learner stu- 
dents who scored 3 or 4 on the U-PASS mathemat- 
ics assessments were lower than the percentages 
of non-English language learner students who 
did. Some 7 percent of English language learner 
students who took the algebra I assessment and 15 
percent who took the geometry assessment scored 
3 or 4, compared with 20 percent of non-English 
language learner students who took the algebra I 
assessment and 42 percent who took the geometry 
assessment (figure 6). 

What are the differences in performance between 
English language learner students and non-English 
language learner students on the U-PASS English 

FIGURE 5 

Distribution of scores on the Utah Performance 
Assessment System for Students English language 
arts assessment, by grade level and English 
language learner status, 2008/09 (percent) 



U-PASS score® 

■ la ■ 1b 2a 2b 3 ■ 4 




Percent 

a. Higher numbers indicate better performance. 

Note; Values may not sum to totals because of rounding. 

Source: Authors' analysis of student demographic data and Utah Perfor- 
mance Assessment System for Students data from the Utah State Office 
of Education. 



language arts and mathematics assessments 
by content standard? English language learner 
students scored lower than non-English language 
learner students on all U-PASS content standards 
(table 4). The normalized differences were 1.3-1. 6 
standard deviations on the English language arts 
standards and 0.4-0. 7 standard deviation on the 
mathematics standards. 

The English language arts analyses show consis- 
tency across grades, with the results for grades 10 
and 11 within 0.1 standard deviation of each other. 
Across both grades, English language learner 
students scored 1.6 standard deviations lower on 
standard I (comprehension), 1.4 standard devia- 
tions lower on standard II (writing), and 1.3 stan- 
dard deviations lower on standard III (inquiry) 
than non-English language learner students did. 

The mathematics analyses show smaller differ- 
ences between English language learner and non- 
English language learner students. Across both 

FIGURE 6 

Distribution of scores on the Utah Performance 
Assessment System for Students mathematics 
assessments, by mathematics course and English 
language learner status, 2008/09 (percent) 



U-PASS score® 

■ la ■ 1b 2a 2b 3 ■ 4 
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Percent 



a. Higher numbers indicate better performance. 

Note: Values may not sum to totals because of rounding. 

Source: Authors' analysis of student demographic data and Utah Perfor- 
mance Assessment System for Students data from the Utah State Office 
of Education. 
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TABLE 4 

Difference in average number of correct answers on the U-PASS by content standard and English language 
learner status, 2008/09 



English Non-English 

language learner language learner 

students students 

U-PASS Mean score Mean score 



content area 
and level 


Content 

standard 3 


Total items 


(percent 

correct) 


Standard 

deviation 


(percent 

correct) 


Standard 

deviation 


Absolute 

difference 


Normalized 

difference 15 


English language arts 


Grade 10 


Standard 1 


42 


18.8(44.8) 


9.9 


32.8 (78.1) 


9.0 


14.0 


1.6 




Standard II 


18 


7.1 (39.5) 


4.0 


12.7 (70.3) 


4.1 


5.5 


1.4 




Standard III 


10 


4.4 (43.8) 


2.6 


7.2 (72.3) 


2.2 


2.8 


1.3 


Grade 11 


Standard 1 


42 


16.4 (39.0) 


9.6 


31.7 (75.6) 


9.6 


15.4 


1.6 




Standard II 


18 


7.2 (40.1) 


4.3 


13.1 (72.8) 


4.2 


5.9 


1.4 




Standard III 


10 


3.9 (39.4) 


2.5 


7.2 (71.7) 


2.4 


3.2 


1.3 


Mathematics 


Algebra 1 


Standard 1 


11 


3.5 (31.5) 


2.4 


4.6 (42.0) 


2.7 


1.1 


0.4 




Standard II 


18 


4.9 (27.1) 


3.3 


6.5 (35.8) 


4.1 


1.6 


0.4 




Standard III 


30 


8.0 (26.9) 


5.2 


10.5 (35.4) 


6.3 


2.5 


0.4 




Standard IV 


11 


4.9 (45.6) 


2.8 


6.4 (60.0) 


3.2 


1.5 


0.5 


Geometry 


Standard 1 


33 


13.8(41.8) 


6.7 


18.8 (57.2) 


7.3 


5.0 


0.7 




Standard II 


12 


3.8 (32.1) 


2.4 


5.0(41.9) 


2.8 


1.2 


0.4 




Standard III 


13 


4.4 (34.7) 


2.8 


5.8 (45.1) 


3.2 


1.4 


0.4 




Standard IV 


12 


4.8 (39.8) 


2.9 


6.5 (54.1) 


3.0 


1.7 


0.6 



a. See box 2 for a description of each standard. 

b. The normalized difference is calculated as the absolute difference divided by the pooled standard deviation of both groups (see box 3). 

Note: Of the 2,975 English language learner students who took the U-PASS English language arts assessment in 2008/09, 1,691 were in grade 10 and 1,284 in 
grade 11; of the 68,188 non-English language learner students who took the U-PASS English language arts assessment in 2008/09, 34,635 were in grade 10 
and 33,553 were in grade 11. Of the 2,081 English language learner students who took a U-PASS mathematics assessment in 2008/09, 1,183 took the algebra 
I assessment and 898 took the geometry assessment; of the 24,621 non-English language learner students who took a U-PASS mathematics assessment in 
2008/09, 8,407 took the algebra I assessment and 16,214 took the geometry assessment. 

Source: Authors' analysis of student demographic data and Utah Peformance Assessment System for Students data from the Utah State Office of Education. 



algebra I and geometry, English language learner 
students scored at least 0.4 standard deviation 
lower than non-English language learner students 
on all content standards. The largest difference, 0.7 
standard deviation, was in standard I in geometry 
(algebraic, spatial, and logical reasoning). 

What are the differences in performance on the 
U-PASS English language arts and mathematics 
assessments between English language learner 
students and non-English language learner 
students by content standard ? Because of policy- 
makers’ interest in the performance of students 



who demonstrate English language proficiency 
on the UALPA, the analysis of student assessment 
scores was repeated omitting the scores of English 
language learner students who scored below ad- 
vanced on the UALPA — that is, the scores of non- 
English language learner students were compared 
only with the scores of English language learner 
students who scored advanced on the UALPA. 
Utah State Office of Education staff requested 
this analysis to explore performance disparities 
between English language learner students who 
were demonstrating language proficiency and 
non-English language learner students. 
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On the U-PASS English language arts assessment, 
English language learner students who scored 
advanced on the UALPA scored lower than non- 
English language learner students (figure 7). The 
median scaled score for English language learner 
students who scored advanced on the UALPA 
was 161 in grade 10 and 157 in grade 11, com- 
pared with 170 for non-English language learner 
students in grade 10 and 169 for those in grade 11. 
These gaps are smaller than those in the com- 
parison of all English language learner students 
and non-English language learner students (see 
figure 3). 

Similar results were obtained for mathematics. 

On the U-PASS algebra I and geometry assess- 
ments, English language learner students who 
scored advanced on the UALPA scored lower than 



non-English language learner students (figure 
8). The median scaled score for English language 
learner students who scored advanced on the 
UALPA was 148 in algebra I and 152 in geometry, 
compared with 152 for non-English language 
learner students in algebra and 158 for those in 
geometry. Again, these gaps are smaller than those 
in the comparison of all English language learner 
students and non-English language learner stu- 
dents (see figure 4). 

The percentages of English language learner 
students who scored advanced on the UALPA 
and who scored 3 or 4 (the highest two scores) 
on the U-PASS English language arts assess- 
ment were lower than the percentages of non- 
English language learner students who scored 
3 or 4 on the U-PASS English language arts 



FIGURE 7 

Distribution of scaled scores on the Utah 
Performance Assessment System for Students 
English language arts assessment among English 
language learner students who scored advanced 
on the Utah Academic Language Proficiency 
Assessment and non-English language learner 
students, by grade level, 2008/09 
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Note.- The scale score ranges from 130 to 199. 

Source: Authors' analysis of student demographic data and Utah Perfor- 
mance Assessment System for Students data from the Utah State Office 
of Education. 



FIGURE 8 

Distribution of scaled scores on the Utah 
Performance Assessment System of Students 
mathematics assessment among English 
language learner students who scored advanced 
on the Utah Academic Language Proficiency 
Assessment and non-English language learner 
students, by mathematics course, 2008/09 
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(N = 300) {N = 360) 

ALGEBRA I GEOMETRY 

Note: Counts are greater for geometry because most grade 10 and 
11 non-English language learner students taking mathematics have 
already taken algebra I. The scale score ranges from 130 to 199. 

Source: Authors' analysis of student demographic data and Utah Perfor- 
mance Assessment System for Students data from the Utah State Office 
of Education. 
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assessment (figure 9). Among English language 
learner students who scored advanced on the 
UALPA, 59 percent in grade 10 and 33 percent in 
grade 11 scored 3 or 4 on the U-PASS English lan- 
guage arts assessment, compared with 86 percent 
of non-English language learner students in grade 
10 and 83 percent in grade 11. 

The percentages of English language learner 
students who scored advanced on the UALPA and 
scored 3 or 4 on the U-PASS mathematics assess- 
ments were lower than the percentages of non- 
English language learner students who scored 3 
or 4 on the U-PASS mathematics examinations 
(figure 10). But these differences are smaller than 
those on the English language arts assessment. Of 
the English language learner students who scored 
advanced on the UALPA, 12 percent who took 



the algebra I assessment and 23 percent who took 
the geometry assessment scored 3 or 4, compared 
with 20 percent of non-English language learner 
students who took the algebra I assessment and 42 
percent who took the geometry assessment. 

Differences by content standard between the 
performance of English language learner stu- 
dents who scored advanced on the UALPA and 
their non-English language learner peers were 
also compared. At grade 10, the gaps in English 
language arts performance — 0.8 standard devia- 
tion in comprehension, 0.8 standard deviation in 
writing, and 0.6 standard deviation in inquiry — 
were half as large as those in the analysis that in- 
cluded all English language learner students — 1.6 
standard deviations in comprehension, 1.4 
standard deviations in writing, and 1.3 standard 



FIGURE 9 

Distribution of scores on the Utah Performance 
Assessment System for Students English language 
arts assessment among English language learner 
students who scored advanced on the UALPA and 
non-English language learner students, by grade 
level, 2008/09 
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a. Higher numbers indicate better performance. 

Source: Authors' analysis of Utah Academic Language Proficiency As- 
sessment and Utah Performance Assessment System for Students data 
from the Utah State Office of Education. 



FIGURE 10 

Distribution of scores on the Utah Performance 
Assessment System for Students mathematics 
assessments among English language learner 
students who scored advanced on the Utah 
Academic Language Proficiency Assessment 
and non-English language learner students, by 
mathematics course, 2008/09 



U-PASS score 3 
■ la Bib 2a 2b 3 




25 50 75 100 

Percent 



a. Higher numbers indicate better performance. 

Note: Values may not sum to totals because of rounding. Counts are 
greater for geometry because most grade 10 and 11 non-English lan- 
guage learner students taking mathematics have already taken algebra I. 

Source: Authors' analysis of Utah Academic Language Proficiency As- 
sessment and Utah Performance Assessment System for Students data 
from the Utah State Office of Education. 
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deviations in inquiry (table 5). At grade 11, the 
gaps — 1.2 standard deviations in comprehension, 
1.0 standard deviation in writing, and 1.0 stan- 
dard deviation in inquiry — were about a quarter 
smaller than those in the analysis that included all 
English language learner students — 1.6 standard 
deviations in comprehension, 1.4 standard devia- 
tions in writing, and 1.3 standard deviations in 
inquiry. In both analyses, the largest gaps be- 
tween English language learner and non-English 
language learner students were in comprehension 



(standard I), and the lowest were in the inquiry 
(standard III). 

In mathematics, English language learner students 
who scored advanced on the UALPA outscored 
non-English language learner students by 0.2-0.4 
standard deviation, depending on the examina- 
tion and standard. On all four algebra I standards, 
the difference was 0.2 standard deviation, and on 
all four geometry standards, the difference was 
0.3-0.4 standard deviation. 



TABLE 5 

Difference in mean correct answers on the Utah Performance Assessment System for Students among 
English language learner students who scored advanced on the UALPA and non-English language learner 
students, by content standard and grade level or mathematics course, 2008/09 



English language learner Non-English 

students who scored language learner 

advanced on the UALPA students 

U-PASS Mean score Mean score 



content area 
and level 


Content 

standard 3 


Total items 


(percent 

correct) 


Standard 

deviation 


(percent 

correct) 


Standard 

deviation 


Absolute 

difference 


Normalized 

difference 13 


English language arts 




Standard 1 


42 


25.6 (60.9) 


8.3 


32.8 (78.1) 


9.0 


7.2 


0.8 


Grade 10 


Standard II 


18 


9.3 (51.6) 


3.6 


12.7 (70.3) 


4.1 


3.4 


0.8 




Standard III 


10 


5.9 (59.1) 


2.1 


7.2 (72.3) 


2.2 


1.3 


0.6 




Standard 1 


42 


20.0 (47.7) 


9.3 


31.7 (75.6) 


9.6 


11.7 


1.2 


Grade 11 


Standard II 


18 


8.8 (48.7) 


4.1 


13.1 (72.8) 


4.2 


4.3 


1.0 




Standard III 


10 


4.8 (47.7) 


2.3 


7.2 (71.7) 


2.4 


2.4 


1.0 


Mathematics 




Standard 1 


11 


4.2 (38.2) 


2.4 


4.6 (42.0) 


2.7 


0.4 


0.2 


Algebra 1 


Standard II 


18 


5.8 (31.5) 


3.7 


6.5 (35.8) 


4.1 


0.8 


0.2 


Standard III 


30 


9.2 (31.2) 


5.5 


10.5 (35.4) 


6.3 


1.3 


0.2 




Standard IV 


11 


5.7 (53.4) 


2.9 


6.4 (60.0) 


3.2 


0.7 


0.2 




Standard 1 


33 


15.7 (47.6) 


6.4 


18.8 (57.2) 


7.3 


3.1 


0.4 


Geometry 


Standard II 


12 


4.2 (35.8) 


2.5 


5.0(41.9) 


2.8 


0.8 


0.3 


Standard III 


13 


4.9 (39.0) 


2.9 


5.8 (45.1) 


3.2 


0.8 


0.3 




Standard IV 


12 


5.4 (45.4) 


2.9 


6.5 (54.1) 


3.0 


1.1 


0.4 



a. See box 2 for a description of each standard. 

b. The normalized difference is calculated as the absolute difference divided by the pooled standard deviation of both groups (see box 3). 

Note: Of the 903 English language learner students who scored advanced on the UALPA and who took the U-PASS English language arts assessment in 
2008/09, 488 were in grade 10 and 415 were in grade 11; of the 68,188 non-English language learner students who took the U-PASS English language arts 
assessment in 2008/09, 34,635 were in grade 10 and 33,553 were in grade 11. Of the 660 English language learner students who took a U-PASS mathematics 
assessment in 2008/09, 300 took the algebra I assessment and 360 took the geometry assessment; of the 24,621 non-English language learner students who 
took a U-PASS mathematics assessment in 2008/09, 8,407 took the algebra I assessment and 16,214 took the geometry assessment. 

Source: Authors' analysis of student demographic data and Utah Peformance Assessment System for Students data from the Utah State Office of Education. 
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CONCLUSION 

In Utah, roughly 4 percent of students in grades 
10 and 11 are designated as English language 
learner students. All English language learner 
students in the state take the language proficiency 
assessment, the UALPA. Recent changes in the 
requirements for redesignating English language 
learner students — dropping the minimum score 
on the U-PASS English language arts assessment 
by one category, from 3 to 2b — results in more 
students who score advanced on the UALPA being 
redesignated as fluent. The analysis of 2008/09 
data shows a 29 percent gain at grade 10 and 
a 51 percent gain at grade 11, compared with 
the results under the old requirements. English 
language learner students scored 3 or 4 on the 
English language arts content assessment and not 
advanced on the language proficiency assessment 
in 9 percent of grade 10 cases and 5 percent of 
grade 11 cases. These figures support the anec- 
dotal reports of this counterintuitive pairing of 
results. 

English language learner students, including those 
who scored advanced on the UALPA, scored lower 
than non-English language learner students on 
the U-PASS overall as well as on each content stan- 
dard. In English language arts, English language 
learner students were outscored by non-English 
language learner students by 1.3-1. 6 standard 
deviations, a pattern that was consistent across 
grades: in grades 10 and 11, English language 



English language learner 
students, including those 
who scored advanced 
on the UALPA, scored 
lower than non-English 
language learner 
students on the U-PASS 
overall as well as on 
each content standard 



learner students were 
outscored by non-Eng- 
lish language learner 
students by 1.6 standard 
deviations on standard 
I (comprehension), 1.4 
standard deviations on 
standard II (writing), and 
1.3 standard deviations 
on standard III (inquiry). 

Although neither the 

content of the standards nor the items in the 
standards were reviewed, the magnitude and con- 
sistency of the results suggest that such a review 
by curriculum experts might illuminate specific 
drivers of the gap in performance between English 
language learner and non-English language 
learner students. In mathematics, the differences 
were smaller, with English language learner stu- 
dents outscored by non-English language learner 
students by 0.4-0.7 standard deviation. 



When only English language learner students who 
scored advanced on the UALPA are compared to 
non-English language learner students, the differ- 
ences by content standard on the U-PASS English 
language arts exam are halved for grade 10 and 
reduced by a quarter for grade 11. Differences by 
content standard on the U-PASS mathematics 
assessments fall to 0.2-0.4 standard deviation 
— or about half the differences when all English 
language learner students are compared with non- 
English language learner students. 
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